Lidar point selection using image segmentation

ABSTRACT

The subject disclosure relates to techniques for selecting points of an image for processing with LiDAR data. A process of the disclosed technology can include steps for receiving an image comprising a first image object and a second image object, processing the image to place a bounding box around the first image object and the second image object, and processing an image area within the bounding box to identify a first image mask corresponding with a first pixel region of the first image object and a second image mask corresponding with a second pixel region of the second image object. Systems and machine-readable media are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation and claims the benefit of U.S.application Ser. No. 16/815,123, filed on Mar. 11, 2020, which isexpressly incorporated by reference herein in its entirety.

BACKGROUND 1. Technical Field

The subject technology provides solutions for facilitating distanceestimations of image objects and in particular, for usingmachine-learning models to segment potentially occluded image objectsfor distance estimation using Light Detection and Ranging (LiDAR) data.

2. Introduction

Image processing systems that are configured to perform objectrecognition often use bounding boxes to identify image regionscorresponding with locations of objects of interest. In someapplications, objects need to be transformed into three-dimensional (3D)space.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of the subject technology are set forth in the appendedclaims. However, the accompanying drawings, which are included toprovide further understanding, illustrate disclosed aspects and togetherwith the description serve to explain the principles of the subjecttechnology. In the drawings:

FIG. 1 illustrates an example system that can be used to select pointsin images for processing with Light Detection and Ranging (LiDAR) data,according to some aspects of the disclosed technology.

FIG. 2 illustrates an example of an initial bounding box placementperformed using a bounding box placement process of the disclosedtechnology.

FIG. 3 illustrates an example of an image mask placement using an imagemask placement process, according to some aspects of the disclosedtechnology.

FIG. 4 illustrates steps of an example process for selecting points inan image for processing with LiDAR data, according to some aspects ofthe disclosed technology.

FIG. 5 illustrates an example environment that includes an autonomousvehicle in communication with a remote computing system, according tosome aspects of the disclosed technology.

FIG. 6 illustrates an example processor-based system with which someaspects of the subject technology can be implemented.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology can bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a more thoroughunderstanding of the subject technology. However, it will be clear andapparent that the subject technology is not limited to the specificdetails set forth herein and may be practiced without these details. Insome instances, structures and components are shown in block diagramform in order to avoid obscuring the concepts of the subject technology.

As described herein, one aspect of the present technology is thegathering and use of data available from various sources to improvequality and experience. The present disclosure contemplates that in someinstances, this gathered data may include personal information. Thepresent disclosure contemplates that the entities involved with suchpersonal information respect and value privacy policies and practices.

In some image processing techniques, bounding boxes are used to identifyan image region that contains one or more objects (image objects) ofpotential interest. However, using conventional bounding box processingtechniques, it is not uncommon for multiple image objects to overlap intwo-dimensional (2D) pixel space. For example, an object of potentialinterest may be partially occluded by an object of no interest;alternatively, two objects of interest may partially occlude oneanother. Close proximities (or occlusions) between image objects make itdifficult to disaggregate and classify the image objects, as well as toperform object ranging, for example, to determine locations of differentimage objects in three-dimensional (3D) space. Such errors can beespecially problematic for image-processing needed to enable autonomousvehicle (AV) navigation and guidance. For example, autonomous vehiclesrely heavily on thousands of images received from onboard cameras orsensors every minute to navigate on roadways. Each of these imagescapture many objects, such as pedestrians, other vehicles, sidewalks,road signs, etc. However, all of these objects are captured intwo-dimensional (2D) space as images. Thus, there is a need to maximizehighly accurate data during transformation from 2D space into 3D space.In other words, there is a need for facilitating highly accuratedistance estimations of objects in images.

Aspects of the disclosed technology address the limitations ofconventional distance estimations of objects in images by usingmachine-learning models to segment the images and select points withinthe segments of the images for processing with Light Detection andRanging (LiDAR) data. More specifically, one or more neural networks aretrained to classify pixels based on objects in the image at thecorresponding pixel and place bounding boxes around the objects. The oneor more neural networks also segment images into image segments or pixelregions that can be selected. The one or more neural networks may thencombine the bounding boxes and the pixel regions to select specificpixels or points on the image to belong to each object. Then, theselected points of the image can be processed with LiDAR data todetermine the depth of the object at the selected points of the image.

As understood by those of skill in the art, machine-learning basedclassification techniques can vary depending on the desiredimplementation. For example, machine-learning classification schemes canutilize one or more of the following, alone or in combination: hiddenMarkov models, recurrent neural networks (RNNs), convolutional neuralnetworks (CNNs); Deep Learning networks, Bayesian symbolic methods,general adversarial networks (GANs), support vector machines, imageregistration methods, and/or applicable rule-based systems. Whereregression algorithms are used, they can include but are not limited to:a Stochastic Gradient Descent Regressors, and/or Passive AggressiveRegressors, etc.

Machine learning classification models can also be based on clusteringalgorithms (e.g., a Mini-batch K-means clustering algorithm), arecommendation algorithm (e.g., a Miniwise Hashing algorithm, orEuclidean Locality-Sensitive Hashing (LSH) algorithm), and/or an anomalydetection algorithm, such as a Local outlier factor. Additionally,machine-learning models can employ a dimensionality reduction approach,such as, one or more of: a Mini-batch Dictionary Learning algorithm, anIncremental Principal Component Analysis (PCA) algorithm, a LatentDirichlet Allocation algorithm, and/or a Mini-batch K-means algorithm,etc.

FIG. 1 illustrates an example system 100 that can be used to facilitatedistance estimates of objects, according to some aspects of thedisclosed technology. System 100 includes one or more cameras 102 thatcapture and store unprocessed (unbounded) images into a repository ofunclassified images 104 from which the unprocessed images are providedto one or more neural networks 106. Similarly, the system 100 alsoincludes one or more Light Detection and Ranging (LiDAR) instruments 114that record and capture LiDAR data, for example, that is stored in apointcloud database 116. The LiDAR data can represent the same objectsrecorded by cameras 102; that is, the LiDAR data corresponds to theimages captured by the one or more cameras 102.

The one or more neural networks 106 can be configured to receiveunclassified images 104 and to to identify one or more image objects inthe unprocessed images. Neural networks 106 can then place boundingboxes around the one or more image objects, and output the bounded imageobjects to a repository of bounded image objects 108. For example, animage may include a person (i.e. a first image object) occluded by abush (i.e. a second image object); thus, the one or more neural network106 will bound the person with a first bounding box and the bush with asecond bounding box. However, in some instances, parts of the bush mayoverlap the first bounding box and parts of the person may overlap thesecond bounding box.

The one or more neural networks 106 can also be configured to receiveand segment, based upon detected objects, the unbounded image into pixelregions associated with the detected objects. The one or more neuralnetworks 106 can then identify image masks corresponding to the pixelregions. The image masks are then stored in a repository of image masks110.

The bounded image objects and the image masks of interest are combinedto select points 112 for processing with LiDAR data. Selected points 112are then processed 118 with the corresponding LiDAR data stored in thepointcloud 116 to determine distance estimates 120 corresponding to eachselected point in the bounded image objects.

In some instances, image processing 118 may identify masks correspondingto detected objects only within the bounded image objects. In otherwords, the image processing 118 may occur within an image area withinthe bounding box to identify the image masks corresponding to the pixelregion of the bounded image object. In these instances, the totalprocessing power utilized is reduced because the image processing 118 isnot occurring for all objects in the image.

In some implementations, the one or more neural networks 106 can processone or more pixels in the pixel regions to determine classificationlabels for the image objects. The one or more neural networks 106 canthen associate the classification label with the image objects. Examplesof the foregoing embodiments are discussed in relation to graphicalexamples provided by FIGS. 2- 5 , discussed below.

FIG. 2 graphically illustrates placement of bounding boxes 212, 222performed using a process of the disclosed technology. In the example ofFIG. 2 , image 200 can represent an initial output of a bounding boxprocess performed by one or more neural networks. The one or more neuralnetworks are configured to detect one or more image objects, such asvehicles, people, signs, etc. In this example, a first image object 210and a second image object 220 are detected (e.g., using machine-learningtechniques). The first image object 210 is a pedestrian walking and thesecond image object 220 is a vehicle. The one or more neural networksinsert into image 200 a first bounding box 212 encompassing a pixel areaof the first image object 210 and a second bounding box 222 encompassinga pixel area of the second image object 220. As shown, the first imageobject 210 is partially occluded by the second image object 220, suchthat the first bounding box 212 includes a portion 224 of the secondimage object 204 (the pole). In other words, the pedestrian ispositioned behind the pole, partially occluded by the pole. Similarly,the second bounding box 222 includes a portion 214 of the first imageobject 210 (the pedestrian). It is understood that a similar boundingbox processing methodology can be applied to one or more other imageobjects in the same image set.

FIG. 3 illustrates a placement of image masks 312, 322 performed using aprocess of the disclosed technology. In particular, image 300illustrates an example in which image masks 312, 322 are identified inthe image 300. As shown, in some instances, the image masks 312, 322 areidentified solely within bounding boxes 212, 222. As furtherillustrated, a first image mask 312 is identified corresponding with afirst pixel region of the first image object 210 and a second image mask322 is identified corresponding with a second pixel region of the secondimage object 220. The image masks 312, 322 thus more accuratelyencompasses and identify respectively the first and second image objects210, 220. Additionally, as discussed above, additional processing stepscan be performed to identify semantic labels that are associated withthe image object. In the example provided by images 200, 300, thesemantic label “person” may be associated with the first image object210 and the semantic label “car” may be associated with the second imageobject 220.

FIG. 3 further illustrates points (shown as dots) within the image masks312, 322. These points are selected for processing with LiDAR data, suchthat LiDAR data corresponding to the location of the points is used toprocess the points for distance estimates. Thus, each point correspondsto a distance estimate that may be used by an autonomous vehicle. Insome instances, the location of the points may be used to processdistance estimates for each image object 210, 220 as a whole. In otherwords, the output of the points processed with LiDAR data will result ina depth of or distance estimate of the image object 210, 220 instead ofdistance estimates for each individual point.

FIG. 4 illustrates steps of an example process 400 for point selectionfor processing with LiDAR data, according to some aspects of thedisclosed technology. Process 400 begins with step 402 in which an imageis received. More specifically, the image is received from a first dataset recorded by one or more cameras. In some instances, the image may bereceived from a machine-learning model that has processed the image. Inother instances, as discussed above, the image may be an unprocessedimage that contains one or more unbounded image objects. The imagecomprises at least a first image object and a second image object, suchthat at least a portion of the first image object is occluded orobscured by at least a portion of the second image object. For example,a leg of a person (i.e. a portion of the first image object) ispartially occluded by a stop sign (i.e. the second image object).Furthermore, we may be interested in selecting points associated withthe person for processing with LiDAR data. In other words, we may beinterested in determining a distance estimate of the person, but not thestop sign.

For clarity and discussion purposes, the following will discuss aninstance, in which at least a portion of the first image object isobscured by at least a portion of the first second image object, whereinthe first image object is a primary image object of interest for LiDARpoint selection. However, it is to be understood that the first imageobject and the second image object may be used interchangeably to labelor select image objects. In some instances, the opposite may be true, inthat at least a portion of the second image object is occluded orobscured by at least a portion of the first image object and the secondimage object is the image object of interest. In some instances, boththe first and second image objects may occlude or obscure at least aportion of the other image object. Moreover, one of ordinary skill inthe art will understand that any combination of image objects may be ofinterest for LiDAR point selection.

In step 404, the image is processed to identify and place a bounding boxaround the image object of interest (i.e. the first image object) andthe portion of the second image object that partially occludes the firstimage object. In some instances, the image is processed through one ormore neural networks, such that processing the image to place thebounding box around the first image object and the second image objectis performed using a first machine-learning model.

In step 406, the image is processed to identify image maskscorresponding to pixel regions of each image object. Thus, the image isprocessed to identify a first image mask corresponding with a firstpixel region of the first image object and a second image maskcorresponding with a second pixel region of the second image object. Insome instances, only an image area within the bounding box of the imageis processed, such that the overall processing power is reduced. In someinstances, the image is processed through one or more neural networks,such that processing the image or image area within the bounding box toidentify the first image mask and second image masks is performed usinga first machine-learning model.

In step 408, one or more pixels in the pixel regions are processed todetermine classification labels for the image objects. Thus, one or morepixels in the first pixel region are processed to determine a firstclassification label for the first object and one or more pixels in thesecond pixel region are processed to determine a second classificationlabel for the second image object.

In step 410, the classification labels are associated with thecorresponding image objects. In other words, the first classificationlabel is associated with the first image object and the secondclassification label is associated with the second image object.

In step 412, the pixel regions are processed with LiDAR data todetermine a depth of the corresponding image objects. Thus, the firstpixel region is processed to determine a depth of the first image objectbased on LiDAR data and the second pixel region is processed todetermine a depth of the second image object based on LiDAR data.

In step 414, the image objects are associated with a range or distanceestimate based upon the determined depths, which is based on LiDAR data.Thus, the first image object is associated with a first range ordistance estimate based on LiDAR data and the second image object isassociated with a second range or distance estimate based on LiDAR data.

FIG. 5 illustrates environment 500 that includes an autonomous vehicle502 in communication with a remote computing system 550.

Autonomous vehicle 502 can navigate about roadways without a humandriver based upon sensor signals output by sensor systems 504-506 of theautonomous vehicle 502. The autonomous vehicle 502 includes a pluralityof sensor systems 504-506 (a first sensor system 504 through an Nthsensor system 506). The sensor systems 504-506 are of different typesand are arranged about the autonomous vehicle 502. For example, thefirst sensor system 504 may be a camera sensor system, and the Nthsensor system 506 may be a lidar sensor system. Other exemplary sensorsystems include radar sensor systems, global positioning system (GPS)sensor systems, inertial measurement units (IMU), infrared sensorsystems, laser sensor systems, sonar sensor systems, and the like.

The autonomous vehicle 502 further includes several mechanical systemsthat are used to effectuate appropriate motion of the autonomous vehicle502. For instance, the mechanical systems can include but are notlimited to, a vehicle propulsion system 530, a braking system 532, and asteering system 534. The vehicle propulsion system 530 may include anelectric motor, an internal combustion engine, or both. The brakingsystem 532 can include an engine brake, brake pads, actuators, and/orany other suitable componentry that is configured to assist indecelerating the autonomous vehicle 502. The steering system 534includes suitable componentry that is configured to control thedirection of movement of the autonomous vehicle 502 during navigation.

The autonomous vehicle 502 further includes a safety system 536 that caninclude various lights and signal indicators, parking brake, airbags,etc. The autonomous vehicle 502 further includes a cabin system 538 thatcan include cabin temperature control systems, in-cabin entertainmentsystems, etc.

The autonomous vehicle 502 additionally comprises an internal computingsystem 510 that is in communication with the sensor systems 504-506 andthe systems 530, 532, 534, 536, and 538. The internal computing systemincludes at least one processor and at least one memory havingcomputer-executable instructions that are executed by the processor. Thecomputer-executable instructions can make up one or more servicesresponsible for controlling the autonomous vehicle 502, communicatingwith remote computing system 550, receiving inputs from passengers orhuman co-pilots, logging metrics regarding data collected by sensorsystems 504-506 and human co-pilots, etc.

Internal computing system 510 can include control service 512 that isconfigured to control the operation of vehicle propulsion system 530,braking system 532, steering system 534, safety system 536, and cabinsystem 538. The control service 512 receives sensor signals from thesensor systems 504-506 as well communicates with other services of theinternal computing system 510 to effectuate operation of the autonomousvehicle 502. In some embodiments, control service 512 may carry outoperations in concert one or more other systems of autonomous vehicle502.

The internal computing system 510 can also include a constraint service514 to facilitate safe propulsion of the autonomous vehicle 502. Theconstraint service 514 includes instructions for activating a constraintbased on a rule-based restriction upon operation of the autonomousvehicle 502. For example, the constraint may be a restriction uponnavigation that is activated in accordance with protocols configured toavoid occupying the same space as other objects, abide by traffic laws,circumvent avoidance areas, etc. In some embodiments, the constraintservice can be part of the control service 512.

The internal computing system 510 can also include a communicationservice 516. The communication service can include both software andhardware elements for transmitting and receiving signals from/to theremote computing system 550. The communication service 516 is configuredto transmit information wirelessly over a network, for example, throughan antenna array that provides personal cellular (long-term evolution(LTE), 3G, 5G, etc.) communication.

In some embodiments, one or more services of the internal computingsystem 510 are configured to send and receive communications to remotecomputing system 550 for such reasons as reporting data for training andevaluating machine learning algorithms, requesting assistance fromremoting computing system or a human operator via remote computingsystem 550, software service updates, ridesharing pickup and drop offinstructions etc.

The internal computing system 510 can also include a latency service518. The latency service 518 can utilize timestamps on communications toand from the remote computing system 550 to determine if a communicationhas been received from the remote computing system 550 in time to beuseful. For example, when a service of the internal computing system 510requests feedback from remote computing system 550 on a time-sensitiveprocess, the latency service 518 can determine if a response was timelyreceived from remote computing system 550 as information can quicklybecome too stale to be actionable. When the latency service 518determines that a response has not been received within a threshold, thelatency service 518 can enable other systems of autonomous vehicle 502or a passenger to make necessary decisions or to provide the neededfeedback.

The internal computing system 510 can also include a user interfaceservice 520 that can communicate with cabin system 538 in order toprovide information or receive information to a human co-pilot or humanpassenger. In some embodiments, a human co-pilot or human passenger maybe required to evaluate and override a constraint from constraintservice 514, or the human co-pilot or human passenger may wish toprovide an instruction to the autonomous vehicle 502 regardingdestinations, requested routes, or other requested operations.

As described above, the remote computing system 550 is configured tosend/receive a signal from the autonomous vehicle 502 regardingreporting data for training and evaluating machine learning algorithms,requesting assistance from remote computing system 550 or a humanoperator via the remote computing system 550, software service updates,rideshare pickup and drop off instructions, etc.

The remote computing system 550 includes an analysis service 552 that isconfigured to receive data from autonomous vehicle 502 and analyze thedata to train or evaluate machine learning algorithms for operating theautonomous vehicle 502. The analysis service 552 can also performanalysis pertaining to data associated with one or more errors orconstraints reported by autonomous vehicle 502.

The remote computing system 550 can also include a user interfaceservice 554 configured to present metrics, video, pictures, soundsreported from the autonomous vehicle 502 to an operator of remotecomputing system 550. User interface service 554 can further receiveinput instructions from an operator that can be sent to the autonomousvehicle 502.

The remote computing system 550 can also include an instruction service556 for sending instructions regarding the operation of the autonomousvehicle 502. For example, in response to an output of the analysisservice 552 or user interface service 554, instructions service 556 canprepare instructions to one or more services of the autonomous vehicle502 or a co-pilot or passenger of the autonomous vehicle 502.

The remote computing system 550 can also include a rideshare service 558configured to interact with ridesharing application 570 operating on(potential) passenger computing devices. The rideshare service 558 canreceive requests to be picked up or dropped off from passengerridesharing app 570 and can dispatch autonomous vehicle 502 for thetrip. The rideshare service 558 can also act as an intermediary betweenthe ridesharing app 570 and the autonomous vehicle wherein a passengermight provide instructions to the autonomous vehicle to 102 go around anobstacle, change routes, honk the horn, etc.

As described herein, one aspect of the present technology is thegathering and use of data available from various sources to improvequality and experience. The present disclosure contemplates that in someinstances, this gathered data may include personal information. Thepresent disclosure contemplates that the entities involved with suchpersonal information respect and value privacy policies and practices.

FIG. 6 shows an example of computing system 600, which can be forexample any computing device making up internal computing system 510,remote computing system 550, (potential) passenger device executingrideshare app 570, or any component thereof in which the components ofthe system are in communication with each other using connection 605.Connection 605 can be a physical connection via a bus, or a directconnection into processor 610, such as in a chipset architecture.Connection 605 can also be a virtual connection, networked connection,or logical connection.

In some embodiments, computing system 600 is a distributed system inwhich the functions described in this disclosure can be distributedwithin a datacenter, multiple data centers, a peer network, etc. In someembodiments, one or more of the described system components representsmany such components each performing some or all of the function forwhich the component is described. In some embodiments, the componentscan be physical or virtual devices.

Example system 600 includes at least one processing unit (CPU orprocessor) 610 and connection 605 that couples various system componentsincluding system memory 615, such as read-only memory (ROM) 620 andrandom access memory (RAM) 625 to processor 610. Computing system 600can include a cache of high-speed memory 612 connected directly with, inclose proximity to, or integrated as part of processor 610.

Processor 610 can include any general purpose processor and a hardwareservice or software service, such as services 632, 634, and 636 storedin storage device 630, configured to control processor 610 as well as aspecial-purpose processor where software instructions are incorporatedinto the actual processor design. Processor 610 may essentially be acompletely self-contained computing system, containing multiple cores orprocessors, a bus, memory controller, cache, etc. A multi-core processormay be symmetric or asymmetric.

To enable user interaction, computing system 600 includes an inputdevice 645, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 600 can also include output device 635, which can be one or moreof a number of output mechanisms known to those of skill in the art. Insome instances, multimodal systems can enable a user to provide multipletypes of input/output to communicate with computing system 600.Computing system 600 can include communications interface 640, which cangenerally govern and manage the user input and system output. There isno restriction on operating on any particular hardware arrangement, andtherefore the basic features here may easily be substituted for improvedhardware or firmware arrangements as they are developed.

Storage device 630 can be a non-volatile memory device and can be a harddisk or other types of computer readable media which can store data thatare accessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs), read-only memory (ROM), and/or somecombination of these devices.

The storage device 630 can include software services, servers, services,etc., that when the code that defines such software is executed by theprocessor 610, it causes the system to perform a function. In someembodiments, a hardware service that performs a particular function caninclude the software component stored in a computer-readable medium inconnection with the necessary hardware components, such as processor610, connection 605, output device 635, etc., to carry out the function.

For clarity of explanation, in some instances, the present technologymay be presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

Embodiments within the scope of the present disclosure may also includetangible and/or non-transitory computer-readable storage media ordevices for carrying or having computer-executable instructions or datastructures stored thereon. Such tangible computer-readable storagedevices can be any available device that can be accessed by a generalpurpose or special purpose computer, including the functional design ofany special purpose processor as described above. By way of example, andnot limitation, such tangible computer-readable devices can include RAM,ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storageor other magnetic storage devices, or any other device which can be usedto carry or store desired program code in the form ofcomputer-executable instructions, data structures, or processor chipdesign. When information or instructions are provided via a network oranother communications connection (either hardwired, wireless, orcombination thereof) to a computer, the computer properly views theconnection as a computer-readable medium. Thus, any such connection isproperly termed a computer-readable medium. Combinations of the aboveshould also be included within the scope of the computer-readablestorage devices.

Computer-executable instructions include, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. Computer-executable instructions also includeprogram modules that are executed by computers in stand-alone or networkenvironments. Generally, program modules include routines, programs,components, data structures, objects, and the functions inherent in thedesign of special-purpose processors, etc. that perform tasks orimplement abstract data types. Computer-executable instructions,associated data structures, and program modules represent examples ofthe program code means for executing steps of the methods disclosedherein. The particular sequence of such executable instructions orassociated data structures represents examples of corresponding acts forimplementing the functions described in such steps.

Other embodiments of the disclosure may be practiced in networkcomputing environments with many types of computer systemconfigurations, including personal computers, hand-held devices,multi-processor systems, microprocessor-based or programmable consumerelectronics, network PCs, minicomputers, mainframe computers, and thelike. Embodiments may also be practiced in distributed computingenvironments where tasks are performed by local and remote processingdevices that are linked (either by hardwired links, wireless links, orby a combination thereof) through a communications network. In adistributed computing environment, program modules may be located inboth local and remote memory storage devices.

The various embodiments described above are provided by way ofillustration only and should not be construed to limit the scope of thedisclosure. For example, the principles herein apply equally tooptimization as well as general improvements. Various modifications andchanges may be made to the principles described herein without followingthe example embodiments and applications illustrated and describedherein, and without departing from the spirit and scope of thedisclosure. Claim language reciting “at least one of” a set indicatesthat one member of the set or multiple members of the set satisfy theclaim.

What is claimed is:
 1. A computer-implemented method, comprising:receiving, from a first data set recorded by one or more cameras, animage comprising one or more pixels making up a first image object and asecond image object, wherein the first image object is partiallyoccluded by the second image object; processing the image to place abounding box around the first image object; processing the one or morepixels to determine a depth of the first image object and a depth of thesecond image object based on LiDAR data; and classifying the first imageobject and the second image object.
 2. The computer-implemented methodof claim 1, wherein the classifying the first image object and thesecond image object further comprises: associating the first imageobject with a first semantic label.
 3. The computer-implemented methodof claim 2, wherein the first semantic label being used in 3-dimensionaltransformation of the image for controlling operation of an AutonomousVehicle (AV).
 4. The computer-implemented method of claim 1, furthercomprising: processing the one or more pixels within the bounding box toidentify a first image mask corresponding to the first image object, andto identify a second image mask corresponding to the second imageobject.
 5. The computer-implemented method of claim 1, whereinprocessing the image to place the bounding box around the image objectis performed using a first machine-learning model.
 6. Thecomputer-implemented method of claim 4, wherein processing the one ormore pixels within the bounding box to identify the first image mask isperformed using a second machine-learning model.
 7. Thecomputer-implemented method of claim 1, further comprising: associatinga range with the image object based on LiDAR data.
 8. A system,comprising: one or more processors; and a computer-readable mediumcomprising instructions stored therein, which when executed by theprocessors, cause the processors to: receive, from a first data setrecorded by one or more LiDAR cameras outputting LiDAR data, an imagecomprising a first image object and a second image object; process theimage to place a bounding box around the first image object and thesecond image object; process the image in the bounding box to determinea depth of the first image and a depth the second image object based onthe LiDAR data; processing one or more pixels to determine a firstclassification label for the first image object; and processing the oneor more pixels to determine a second classification label for the secondimage object.
 9. The system of claim 8, wherein the one or moreprocessors are configured to execute the computer-readable instructionsto: process the image within the bounding box to identify a first imagemask corresponding with a first pixel region making up the first imageobject and a second image mask corresponding with a second pixel regionmaking up the second image object; and identify one or more first pixelsin the first image object and the second image object using the boundingbox, the first image mask, and the second image mask.
 10. The system ofclaim 8, wherein the one or more processors are configured to executethe computer-readable instructions to: associate the first image objectwith the first classification label, wherein the first classificationlabel is a semantic label.
 11. The system of claim 10, wherein the firstclassification label being used in 3-dimensional transformation of theimage for controlling operation of an Autonomous Vehicle (AV).
 12. Thesystem of claim 8, wherein the one or more processors are configured toexecute the computer-readable instructions to process the image to placethe bounding box around the first image object and the second imageobject is performed using a first machine-learning model.
 13. The systemof claim 9, wherein the one or more processors are configured to executethe computer-readable instructions to process the image within thebounding box to identify the first image mask is performed using asecond machine-learning model.
 14. The system of claim 8, wherein theone or more processors are configured to execute the computer-readableinstructions to: associate a first range with the first image object anda second range with the second image object based on the LiDAR data. 15.A non-transitory computer-readable medium comprising instructions storedtherein, which when executed by one or more processors, causes the oneor more processors to: receive, from a first data set recorded by one ormore LiDAR cameras outputting LiDAR data, an image comprising a firstimage object and a second image object; process the image to place abounding box around the first image object and the second image object;process the image in the bounding box to determine a depth of the firstimage and a depth the second image object based on the LiDAR data;processing one or more pixels to determine a first classification labelfor the first image object; and processing the one or more pixels todetermine a second classification label for the second image object. 16.The non-transitory computer-readable medium of claim 15, wherein the oneor more processors are configured to execute the computer-readableinstructions to: process the image within the bounding box to identify afirst image mask corresponding with a first pixel region making up thefirst image object and a second image mask corresponding with a secondpixel region making up the second image object; and identify one or morefirst pixels in the first image object and the second image object usingthe bounding box, the first image mask, and the second image mask. 17.The non-transitory computer-readable medium of claim 15, wherein the oneor more processors are configured to execute the computer-readableinstructions to: associate the first image object with the firstclassification label, wherein the first classification label is asemantic label.
 18. The non-transitory computer-readable medium of claim17, wherein the first classification label being used in 3-dimensionaltransformation of the image for controlling operation of an AutonomousVehicle (AV).
 19. The non-transitory computer-readable medium of claim15, wherein the one or more processors are configured to execute thecomputer-readable instructions to process the image to place thebounding box around the first image object and the second image objectis performed using a first machine-learning model.
 20. Thenon-transitory computer-readable medium of claim 16, wherein the one ormore processors are configured to execute the computer-readableinstructions to process the image within the bounding box to identifythe first image mask is performed using a second machine-learning model.